docs: run Feast offline-to-online pipeline with Spark Operator - #310

Summary

• add a runnable Feast offline-to-online notebook that uses an OLM-installed Spark Operator and a SparkApplication for every Spark batch workload
• move the Spark batch program, KServe model server, and eight example Kubernetes manifests into docs/en/train/guides/assets/feast-offline-to-online-inference/
• document a sparse checkout that downloads the notebook and its required asset bundle together while preserving their relative layout
• use copyable %%bash command cells and direct kubectl apply pipelines to create and submit Kubernetes resources; remove the notebook's Python kubectl wrapper and embedded source/YAML strings
• generate the sample's synthetic feature history with Spark and store partitioned Parquet in S3-compatible object storage
• use PostgreSQL only for the durable Feast SQL registry and Redis for materialized online features
• run Feast's Spark point-in-time historical join, sample model training, and Redis materialization inside the SparkApplication driver
• configure S3A through the public SparkSession.builder.config() API before context creation; use Secret-backed AWS environment variables through S3A's credential chain instead of private Py4J APIs or credentials in Spark configuration
• resolve the global-cluster registry through kube-public/global-info and require an approved Spark+Feast runtime instead of embedding an internal registry hostname
• keep S3 credentials in the Spark workload only; the model PVC receives a serving-only Feast configuration
• deploy Redis-backed online inference through KServe
• evolve C16 e2e coverage from a Kubernetes Job and local FileSource to the same SparkApplication + S3 + PostgreSQL registry + Redis architecture, reusing the notebook's batch.py and server.py assets to avoid source drift

Why

For production-sized offline feature history, S3-compatible object storage plus operator-managed Spark is a better scaling boundary than using PostgreSQL as the feature-history store. PostgreSQL remains a good durable registry backend, while Redis remains the low-latency online store. Feast's Spark offline store is a contributed integration without full test coverage, so the notebook calls out the need to qualify it or use a fully supported warehouse for stricter production requirements.

Keeping source files and manifests under assets/ makes the example inspectable, reusable from e2e, and easier to customize. The notebook remains the walkthrough and uses ordinary shell command blocks for resource submission.

Validation

• yarn lint (0 errors and 0 warnings)
• commit-time documentation lint (0 errors and 0 warnings)
• python3 -m py_compile for the shared batch.py and server.py assets
• YAML parsing for all eight shared manifest files
• Bash syntax validation for all six notebook command cells
• bash -n e2e/lib.sh e2e/cases/c16_feast_offline_online.sh
• assertions that the notebook contains no Python kubectl wrapper, embedded batch_source / server_source, C16 path, internal registry hostname, _jsc, or _jvm
• git diff --check
• C16 guard path verified against demo-x86: exits with skip code 77 when sparkapplications.sparkoperator.k8s.io is absent
• prior live demo-x86 validation in mlops-demo-e2e: PostgreSQL registry, SeaweedFS S3 upload, 240-row historical retrieval, Redis materialization, and online reads for two entities

Live installation status

The uploaded OLM package is expected on the x86 cluster, but its API endpoint timed out on repeated direct and proxy attempts during this update. The reachable demo-x86 cluster does not expose the uploaded Spark package in PackageManifest and already contains a different non-OLM Apache Spark operator using the spark.apache.org API group, so no replacement or conflicting second global controller was installed there. The operator installation and full C16 execution remain to be rerun when the x86 API is reachable.